Method and apparatus for pattern matching for intrusion detection/prevention systems

ABSTRACT

A system, method, apparatus and mechanism for estimating worst-case time complexity of a regular expression defining a pattern adapted for identifying malicious packets and comprising one or more back-references (backref-regex) by constructing a non-deterministic finite automaton (NFA) corresponding to the backref-regex (backref-NFA), wherein the backref-NFA comprises a plurality of NFA-states and a respectively labeled edge for each of the one or more back-references of the backref-regex; performing liveness analysis on the backref-NFA to determine for each NFA-state of the backref-NFA a set of back-references alive at the NFA-state; and determining a maximum number of alive back-references over the plurality of NFA-states, wherein the determined maximum number is indicative of the worst-case time complexity of the backref-regex.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of pending U.S. patent applicationSer. No. 12/610,825, filed on Nov. 2, 2009, entitled METHOD ANDAPPARATUS FOR PATTERN MATCHING FOR INTRUSION DETECTION/PREVENTIONSYSTEMS which application claims the benefit of U.S. Provisional PatentApplication Ser. No. 61/197,885, filed Oct. 31, 2008, entitled“TECHNIQUE FOR CONSTRUCTING ROBUST, ATTACK-RESISTANT INTRUSIONDETECTION/PREVENTION SYSTEMS (IDS/IPS)”; both prior applications arehereby incorporated by reference herein in their entireties.

FIELD OF THE INVENTION

The invention relates to the field of communications networks and, morespecifically, to pattern matching by intrusion detection systems (IDSs)and intrusion prevention systems (IPSs).

BACKGROUND OF THE INVENTION

Signature-based intrusion detection systems (IDSs) or intrusionprevention systems (IPSs), such as Snort®, Bro, Cisco SecurityAppliance, Citrix® Application Firewall, and the like, protect a networkby examining headers and content of all packets entering or leaving thenetwork. Such systems raise alerts and/or drop packets upon detectingsuspicious headers or payloads. In general, a suspicious packet isdetected by matching the packet against a database of rules, where eachrule represents a particular signature/pattern of a security exploit.

To represent security exploits as accurately and precisely as possible,the IDS/IPS rule syntax should be sufficiently powerful. Otherwise, alarge number of good packets may be incorrectly marked as harmful, orharmful packets may go undetected. Moreover, the packet processing rateshould keep up with high line speeds without dropping packets orallowing bad packets through. These two goals often conflict because ofthe direct relationship between the expressiveness and complexity of therule language and the packet processing time. One of the common IDS/IPSrule syntaxes is regular expressions. However, unless the rules arewritten with care and the underlying pattern matching is implementedcarefully, processing of a packet may take a long time to complete. Theresulting performance vulnerability may be exploited by an attacker togenerate a low-bandwidth denial-of-service (DoS) attack on the IDS/IPSitself. Such vulnerability can typically be traced back to thebacktracking-based pattern matching of regular expressions.

When only regular expressions define the rules, this vulnerability maybe avoided, for example, by using deterministic matching process.However, increasingly, the rules are written using an extension ofregular expressions with back-references, known as the full regexsyntax, to which known deterministic matching processes cannot beapplied. The backtracking-based pattern matching is generally the onlyoption available for matching regular expressions with back-references(backref-regexes). Accordingly, embedding such backref-regexesexpressions with regular expressions that cause the backtrackingalgorithms to exhibit exponential behavior may cause serious performancevulnerability. Further, unrestricted use of regular expressions, andregular expressions with back-references in particular, makes itchallenging to predict worst-case performance, and therefore, to guardin advance against performance attacks.

To minimize the performance vulnerability, various guidelines have beenoffered, including avoiding back-references and/or backtracking, using amemory-efficient deterministic algorithm for regular expressions, usingonly well-tested regular expressions, avoiding known patterns that incurexponential behavior, and limiting time and memory requirements of thematching phase. However, such guidelines are often inapplicable in thecontext of the IDSs/IPSs. In particular, as new security exploitsappear, IDS/ISP patterns are continually added and updated, primarily bynetwork managers or security professionals, who are not cognizant of theunderlying pattern matching processes. Further, some security exploits,e.g., buffer overflow attacks, are most accurately and preciselyexpressed with complex syntax like full regex. Limiting time or memorymay result in failure to detect bad packets or dropping of harmlesspackets. Therefore limiting the IDS rule syntax, enforcing time ormemory restrictions, and/or relying on the prudence of the rule writersis not appropriate for IDS/IPS applications.

SUMMARY OF THE INVENTION

Various deficiencies in the prior art are addressed through methods andapparatuses for analyzing packets and determining time complexity ofbackref-regexes. In one embodiment, a method for analyzing packetsincludes comparing an input string representing a received packet with apattern defined by a backref-regex. Such a comparison is performed via asingle pass of a non-deterministic finite automaton (backref-NFA)representing the backref-regex, including its back-references.

The input string includes multiple characters representing, for example,values of various fields of the packet. Characters of the string areselected in a sequence, character-by-character, and analyzed until adetermination of a match or no match between the input string and thepattern is made. When a character is selected, a configurations-set,including one or more configurations, is updated. Each of theconfigurations is associated with a respective non-deterministic finiteautomaton (NFA) state of the backref-NFA and adapted to monitor whetherthe character is being matched against a particular back-reference ofthe backref-NFA.

In one embodiment, the configurations-set is updated by addingconfigurations reachable from the one or more configurations of theconfigurations-set without consuming the character, and then determiningall configurations that are reachable, by consuming the character, fromthe configurations in the configurations-set, including the just addedconfigurations. The newly determined configurations form an updatedconfigurations-set, which is used to analyze a next character in thestring. If the string has ended, then the most recent configurations-setis updated with configurations reachable without character consumptionfrom the configurations in the configurations-set and analyzed todetermine whether the configurations-set includes a configurationassociated with a final state. When the final state is found, the matchbetween the string and the pattern is determined. Otherwise, no match ispossible. If at any point, the configurations-set becomes empty, no newcharacters in the string are selected and no-match is possible betweenthe string and the pattern.

One embodiment provides for a method for estimating worst-case timecomplexity of matching a backref-regex. The method includes constructinga backref-NFA corresponding to the backref-regex, where the backref-NFAincludes NFA-states and respectively labeled edges for back-referencesof the backref-regex. Embodiments of the method further includeperforming liveness analysis on the backref-NFA to determine liveback-references for each NFA-state of the backref-NFA. Upon concludingthe liveness analysis, the maximum number of live back-references overall the NFA-states of the backref NFA is determined. Such a maximumnumber is indicative of the time complexity of the backref-regex. Whenmultiple rules are analyzed, each rule that has the time-complexityabove a pre-defined threshold is flagged to a user.

BRIEF DESCRIPTION OF THE DRAWINGS

The teachings of the present invention can be readily understood byconsidering the following detailed description in conjunction with theaccompanying drawings, in which:

FIG. 1 illustrates a high-level diagram of a network environment,according to one embodiment;

FIG. 2A depicts a high-level block diagram of a method for analyzingpackets, according to one embodiment;

FIG. 2B depicts a high-level block diagram of a method for updating aconfigurations-set, according to one embodiment;

FIG. 3 depicts a high-level block diagram of a method for estimating aworst-case complexity of a rule represented by a regular expression withback-references, according to one embodiment;

FIGS. 4A and 4B depict examples of components of an extendednon-deterministic finite automaton (NFA), referred herein asbackref-NFA, according to one embodiment;

FIGS. 5A and 5B depict examples of regular expressions withback-references and corresponding backref-NFAs, according to oneembodiment; and

FIG. 6 depicts a high-level diagram of a computer suitable for use inperforming the functions described herein, according to one embodiment.

To facilitate understanding, identical reference numerals have beenused, where possible, to designate identical elements that are common tothe figures.

DETAILED DESCRIPTION OF THE INVENTION

Robust and fast pattern matching for intrusion detection systems (IDSs)and intrusion prevention systems (IPSs) is disclosed. Methods andapparatuses described herein provide for on-the-fly deterministicmatching of packets against rules (patterns) describing maliciouspackets, where the rules are defined by regular expressions withback-references (backref-regexes) and facilitate predictable, robustbehavior in practice. Further, techniques described herein provide forcompile-time analysis resulting in tight worst-case matching costs,decrease in run time space usage, and improved performance. Rule-by-ruleanalysis of the worst case matching costs allow for automatic detectionand isolation of potentially “bad” rules and facilitate robustperformance in the face of malicious packets.

The description and drawings presented herein merely illustrate theprinciples of the invention. It will be thus appreciated that thoseskilled in the art will be able to devise various arrangements that,although not explicitly described or shown herein, embody the principlesof the invention and are included within its spirit and scope.Furthermore, all examples recited herein are principally intendedexpressly to be only for pedagogical purposes to aid the reader inunderstanding the principles of the invention and the conceptscontributed by the inventors for furthering the art, and are to beconstrued as being without limitation to such specifically recitedexamples and conditions. Moreover, all statements herein recitingprinciples, aspects, and embodiments of the invention, as well asspecific examples thereof, are intended to encompass equivalentsthereof.

FIG. 1 illustrates an example of a network environment in which datapackets are transmitted between an external network and a securedinternal network, according to one embodiment. In a network environment100, packets may be transmitted from an external host 115 over anexternal network 110 and an internal network 130 toward an internal host135 and back. As depicted in FIG. 1, before packets enter or leave theinternal network 130, the packets are validated by an IntrusionDetection System (IDS)/Intrusion Prevention System (IPS) 120.

The IDS/IPS 120 examines packets by comparing headers and/or content ofthe packets with pre-defined rules/policies/filters representing varioussecurity exploits. A rule/policy/filter definition may be based onpacket parameters, such as protocol, source address, destinationaddress, source port, destination port, source interface, destinationinterface, and others, and/or one or more lists of such parameters. Therule/policy/filter may include a set of one or morerules/policies/filters. In one embodiment, the IDS/IPS 120 appliesmultiple pattern matching rules/policies/filters to the incomingpackets. For example, an incoming packet may first be classifiedaccording to header information, and then, by whether the packet matchesa set of representative keywords taken from full rules. Usually, suchactions eliminate most rules from consideration. If some rules remain,the packet is then matched against the remaining rules. In oneembodiment, at least some of such rules are defined usingbackref-regexes.

FIG. 2A depicts a high-level block diagram of a method 200 for analyzingpackets, according to one embodiment. More specifically, the method 200provides for determining whether a packet represented by an input stringis a malicious packet and thus, allows to determine whether the packetshould be accepted, e.g., allowed into/out of an internal network, suchas the internal network 130, or rejected, e.g., dropped, blocked,logged, quarantined, etc. Although primarily depicted and describedherein as being performed sequentially, at least a portion of the stepsof the method 200 may be performed contemporaneously, or in an orderdifferent than depicted and described with respect to FIG. 2A.Furthermore, not all steps depicted in FIG. 2A are always necessary andat least some of the steps may be performed prior to the execution ofthe method 200.

The method 200 begins at step 205 and proceeds to step 210. At step 210,a non-deterministic finite automaton 210 corresponding to abackref-regex 215 is built. In general, the non-deterministic finiteautomaton 210 is a modified NFA that includes a respectiverepresentation for each back-reference of the backref-regex 215. Thenon-deterministic finite automaton 210 referred herein as backref-NFA.

According to one embodiment, patterns/rules adapted to identifymalicious packets are defined by backref-regexes. Formally, syntax fortraditional regular expressions (i.e., without back-references) isdefined by the following grammar:

r=epsilon OR a OR rs OR r|s OR s*OR(s)  (1)

where ‘r’ and ‘s’ are regular expressions built using such a grammar and‘a’ represents a letter. According to one embodiment, to definebackref-regexes, such as the backref-regex 215, formally, the grammar(1) is extended with the following operator: r=\B, where B is a number,specifying the B^(th) bracketed sub-expression.

Methods for constructing an NFA based on the traditional regularexpression are known. For example, one of such methods is described in“Compilers: Principles, Techniques, & Tools”, by Alfred V. Aho, MonicaS. Lam, Ravi Sethi, and Jeffrey D. Ullman, Second Edition,Addison-Wesley, 2007, sections 3.7.2, 3.7.3, pages 159-161, which ishereby incorporated by reference in its entirety (“Aho”). Thetraditional NFA is defined in general by a set of states and anext-state function, which determines its move based on either an inputsymbol, or spontaneously (“epsilon” move/transition). According to oneembodiment, the traditional NFA and NFA construction method are extendedto provide for representations of the backref-regex's back-referencesand incorporating such representations into the backref-NFA.

More specifically, to handle the new operator of the extended syntax,the NFA construction method is modified to allow for additional labelson edges of the NFA. In particular, the following scenarios of regularexpressions are modified:

r:=(s),where this is the k ^(th) bracket  (a)

r:=\B  (b)

When the backref-regex includes scenario (a), the NFA is extended withcomponents shown in FIG. 4A, where additional labels indicate openingand closing brackets of a pair of matching brackets. When thebackref-regex includes scenario (b), the NFA is extended with componentsshown in FIG. 4B, where an additional label denotes a particularback-reference, such as ‘\k.’ At runtime, the expression r is assignedthe exact substring that has already been matched against the k^(th)bracketed sub-expression during the matching process up to the currentpoint.

According to one embodiment, the backref-NFA is formally defined by atuple (B, Q, Q₀, δ, Σ, F), where

-   -   B is the number of back-references, numbered 1 . . . B,    -   Q is a finite set of states,    -   Q₀ is a subset of Q, defining the initial states,    -   Σ is the input alphabet, the set of legal symbols in a word,    -   δ: Q×(Σ∪{ε}∪{(_(k),)_(k), \k:1≦k≦B})→2^(Q) is the transition        relation.    -   F, a subset of Q, is a set of final states

‘(_(k)’ and ‘)_(k)’ are the new transition symbols that indicaterespectively the start and end of the matching brackets k. ‘\k’ is a newtransition symbol that indicates a back-reference k.

In particular, to construct the backref-NFA based on the backref-regex,for each bracketed portion, e.g., ‘(r)’ where r is a regular expression,additional transition edges labeled with notes indicating the openingand closing of a pair of matching brackets embedding the bracketedportion are created. Returning to FIG. 4A, for ease of understanding,assume that r is a regular expression without back-references orbrackets. To construct the backref-NFA for a backref-regex containing apair of brackets k encompassing the regular expression r, first, atraditional NFA for the regular expression ‘r’ is constructed using, forexample, the technique referenced above.

Then, transitional edges 414 ₁ and 414 ₂ are added on both sides of theconstructed NFA and labeled respectively with ‘(_(k)’ and ‘)_(k)’ toidentify the opening and closing brackets of the pair of matchingbrackets k. The transitional edge 414 ₁ provides for anepsilon-transition from a last state 416 of the backref-NFA's portionpreceding the brackets k to a first state of the NFA constructed for‘r’. If the regular expression embedded into the brackets k representsthe last portion of the backref-regex, the transitional edge 414 ₂provides for an epsilon-transition from the last state of the NFA to afinal state 418.

Further, to create a representation for each back-reference of thebackref-regex, e.g., ‘\k’, a respective additional transition edgelabeled with a ‘match’ note is created in the backref-NFA. Returning toFIG. 4B, consider the following expression: ‘r\k’, where r is a regularexpression. To construct the backref-NFA for this expression, first, anNFA for the regular expression ‘r’ is constructed using one of abovediscussed techniques. Then, a transitional edge 424 is created andlabeled ‘\k’ to denote the back-reference. The transitional edge 424 isadded to connect a last state 426 of a backref-NFA's portion precedingthe brackets k, e.g., backref-NFA representing the expression R, to afinal state 428. To transition between the state 426 and final state 428during the run of the backref-NFA, a match for the back-reference k mustbe determined.

Note that though in the above described examples r is a regularexpression without brackets or back-references, similar techniques areapplicable when r is a full regex that includes brackets and/orback-references. In general, there is no limit on the number of levelsof brackets and/or back-references that the backref-regex r may contain.In one embodiment, when the backref-regex is a multi-level expression,the above described techniques are applied level-by-level, beginningwith lower levels and moving up towards the most general level. This maybe done in sequence for each of the back-references or in parallel.

FIGS. 5A and 5B provide specific examples of backref-regexes andcorresponding backref-NFAs. More specifically, FIG. 5A shows abackref-regex 505 ₁ defined as follows, ‘(ab)\1 (cd)\2’. Thebackref-regex 505 ₁ includes two bracket pairs, each containing aregular expression and two back-references, one after each bracket pair.A string including substring ‘ababcdcd’ matches such an expression. Theregular expressions within the bracket pairs do not includeback-references, and thus, standard NFAs, items 510 ₁ and 515 ₁respectively may be constructed in accordance with the techniquesdescribed above. Each such NFA includes NFA-states that, in general,define paths within the NFA that are followed during the run todetermine a match. Typically, to transition between two NFA-statesduring the run, a particular character or expression associated with anedge between the two NFA-states must be present in an input string at aproper position. For example, character ‘a’ is required to transitionfrom the NFA-state q₁ to the NFA-state q₂. However, as discussed above,some transitions are epsilon transitions, i.e., transition that do notconsume a character of the input string.

After the NFAs 510 ₁ and 515 ₁ are constructed, edges are added on bothsides of each of the NFAs 510 ₁ and 515 ₁ and labeled to identifyrespectively the opening and closing brackets. For example, an edgebetween NFA-states q₀ and q₁ represents the opening bracket and an edgebetween NFA-states q₃ and q₄ represents the closing bracket. NFA-stateq₀ is an initial state. Furthermore, additional edges labeled with matchnotes are created to represent the back-references of backref-regex 505₁. For example, an edge between NFA-states q₄ and q₅ that is labeledwith back-reference ‘\1’ is constructed. In one embodiment, totransition from the NFA-state q₄ and to the NFA-state q₅, a match forthe back-reference 2 must be determined. In other words, a substring‘cd’ must be followed by another substring ‘cd’ to transition from theNFA-state q₄ and to the final NFA-state q₅.

The backref-regex of FIG. 5B is similar to the backref-regex of FIG. 5Awith one exception. The backref-regex 505 ₁ of FIG. 5A requires eachcharacter of the backref-regex 505 ₁ to be present in an input stringfor the match to occur. On other hand, for the input string to match thebackref-regex 505 ₂ of FIG. 5B, only one of the two portions of thebackref-regex 505 ₂ is required to be present in the string, where theportions are separated by the vertical line. Accordingly, stringscontaining either of the following substrings would match thebackref-regex 505 ₂: ‘abab’ or ‘cdcd.’ The backref-NFA 520 ₂ isconstructed in a manner similar to the one described above regarding thebackref-NFA 520 ₁, where two alternative paths in the graph shown inFIG. 5B provide for the two possible substrings that can match thebackref-regex 505 ₂, i.e., paths ‘q₀→q₁→q₂→q₃→q₄→q₅’ and‘q₆→q₇→q₈-q₉→q₁₀→q₁₁’.

Returning to FIG. 2A, at step 220, a packet or an input stringrepresenting a packet is received. For example, a packet or its portionmay be viewed as an input string containing multiple characters. Ingeneral, according to method 200, such characters are consecutivelyanalyzed until a determination is made that the packet matches or doesnot match the rule represented by the backref-NFA.

In one embodiment, the method 200 provides for an extension of amatching process for a traditional regular expression to provide formatching of back-references. The traditional matching process works intwo phases:

-   -   First, a regular expression is compiled into an NFA, via        recursion on the structure of the regular expression.    -   Second, on any input string, the matching algorithm scans the        string from left to right, maintaining a set of active        NFA-states. If, upon reaching the end of the string, an        accepting state of the NFA appears in the current active set,        the string is accepted; otherwise, the string is rejected.        This process is on-the-fly process because each character is        scanned at most once.

According to one embodiment, each position in an input string is nowassociated with a configuration, rather than simply with an NFA-state.The set of configurations is referred to as C. In general, aconfiguration records the current NFA-state, substring information foralready scanned brackets, and whether a particular back-reference is inthe process of being matched.

Formally, the configuration is a triple (q, μ, s), where

-   -   q is a NFA-state of the backref-NFA,    -   μ is a pair of partial functions, (μ_(L), μ_(R)), both from [1 .        . . B] to [0 . . . m−1]. For any b, (μ_(L)(b), μ_(R)(b)), if        defined, indicates a substring of the input string which matches        bracket b. ‘⊥’ is used to indicate a totally undefined function.    -   s is either NOMATCH, or MATCH (b, i), where b is a bracket        number in 1 . . . B, and i is an index.

A run of the backref-NFA A=(B, Q, Q₀, δ, Σ, F) on an input string w withlength (i.e., number of characters) m is given by a sequence r=(c₀, i₀);(c₁; i₁); . . . which associates each position of the word with aconfiguration. The sequence must satisfy the following constraints:

-   -   the first position is 0, and the first configuration is initial,        i.e., i₀=0, and c₀=(q, ⊥, NOMATCH), where qεQ₀.    -   successive pairs are related by automaton transitions as        follows. If (c=(q, μ, s), i) and (c′=(q′, μ′, s′), i′) are        successive pairs, then one of the following constraints must        hold        -   (epsilon-transition) q′εδ (q, ε) and i′=i, μ′=μ, and s′=s        -   (input-transition) q′εδ (q, w (i)) and i′=i+1, μ′=μ, and            s′=s        -   (start-bracket) This transition records the start of a            bracket. For some b, q′εδ (q, (_(b)), i′=i, μ_(L)′ is μ_(L)            extended with b mapped to i, and μ_(R)′=μ_(R).        -   (end-bracket) This transition records the end of a bracket.            For some b, q′εδ (q,)_(b)), i′=i, μ_(L)(b)=j, for some j,            and μ_(L)′=μ_(L), μ_(R)′ is μ_(R) extended with b mapped to            i, and s′=s.        -   (begin-matching) s=NOMATCH, μ(b) is defined, q has a single            transition on \b for some k, and s′=MATCH (b, 0), μ′=μ,            q′=q, i′=i        -   (continue-matching) s has the form MATCH (b, p). Let            μ(b)=(l, h). If l+p)<h, and w (l+p)=w (i) then q′=q,            s′=MATCH (b, p+1), i′=i, μ′=μ        -   (end-matching) s has the form MATCH (b, p). Let μ(b)=(l, h).            If (l+p)=h, then q′ is the unique successor of q on \b,            s′=NOMATCH, and i′=i, μ′=μ

Matching of a back-reference is done by a sequence of transitions. The“begin-matching” transition activates matching a back-reference b. The“continue-matching” transition matches the current input character w (i)against the expected character w (l+p) while keeping the state constant.The “end-matching” transition de-activates matching for b and moves theautomaton state forward.

A run r is an accepting run for the input word w of length m if thefinal pair of the run, (c_(n)=(q_(n), μ_(n), s_(n)), i_(n)) is such thatq_(n) is in F and i_(n)=m. The language of a backref-NFA is the set ofinput words/strings for which there is an accepting run.

Returning to FIG. 2A, in general, steps 230 through 270 are performed inaccordance with the above-described formal principles. Morespecifically, at step 230, it is determined whether the matching processis at the end of the input string. If not, at step 235, it is determinedwhether the configurations-set is empty. If at any point, theconfigurations-set becomes empty, the input string cannot be matched tothe pattern defined by the backref-regex. Accordingly, if theconfigurations-set is empty, the method proceeds to step 250, where thestring is rejected and the method 200 finishes with step 270.

However, if the configurations-set is not empty, a next character in theinput string is determined at step 240. In general, characters of theinput string are analyzed in sequence, character-by-character. At step245, the configurations-set is updated based on the determinedcharacter, and the method 200 returns to step 230. As discussed above,generally, the configurations-set contains one or more configurationsfor moving the matching process along one or more paths of thebackref-NFA to determine whether the input string matches the patterndefined by the backref-regex. In general, when the configurations-set isempty, all possible paths have been exhausted and no path for the inputstring exists within the backref-NFA. More specific details of step 245are discussed below with respect to FIG. 2B.

If at step 230 it is determined that all characters of the input stringhave been already considered, the method 200 proceeds to step 255, wherethe configurations-set is updated to account for all possible epsilontransitions from the configurations included in the configurations-set.More detailed explanation of this step is provided below with respect toFIG. 2B.

At step 260, the current configurations-set is analyzed to determinewhether the configurations-set includes a final state, or in otherwords, includes a configuration associated with the final state. If yes,the match between the input string and the pattern defined by thebackref-regex has been detected, and thus, at step 265, the string isaccepted. Otherwise, the string is rejected, i.e., the string does notmatch the pattern defined by the backref-regex.

FIG. 2B depicts a high-level block diagram of a method for updating aconfigurations-set, according to one embodiment. As described above, theconfigurations-set is updated at different points of the matchingprocess employed by the method 200, i.e., steps 255 and 245. In general,one main difference between the two updates is that the update of step255 includes only update based on possible epsilon transitions, whilethe update of step 245 includes an additional update that is based onpossible transitions by consuming the currently selected character. InFIG. 2B, all shown steps are the steps that form step 245. Step 255includes only steps 274 and 276.

More specifically, input for the method 245 includes a currentconfigurations-set that includes one or more configurations and acharacter currently analyzed according to method 200. At step 274, allconfigurations that are reachable from the current configurations-setwithout consuming the character are determined. In other words, eachconfiguration that may be reached by one or more epsilon transition fromthe NFA-state associated with the configuration is determined. Forexample, in FIG. 5A, a configuration associated with state q₁ is reachedfrom configuration associated with state q₀ in such a manner. Similarly,in FIG. 5B, a configuration associated with state q₁₂ is reached from aconfiguration associated with state q₁ without consuming a character ofthe string.

At step 276, each configuration determined at step 274 is added into theconfigurations-set. If at step 274 no configurations were found, nonewill be added.

At step 278, a fresh configuration-set is determined. Specifically, thefresh configuration-set is initially is empty, and it is added-to asfollows. For each configuration C in the current configurations-set,those configurations that are reachable from C by consuming the currentcharacter (as defined by the automaton structure) are determined andadded into the fresh configurations-set. Note that it is possible for aconfiguration to have no reachable configurations on a given character.It should also be noted that it is possible for some of the newlyderived configurations to be identical with those in the current set.

For example, in FIG. 5A, if the configurations-set includes aconfiguration associated with state q1 and the current character is ‘a’,then a configuration that can be reached by consuming ‘a’ will be addedto the configurations-set, i.e., a configuration associated with stateq2.

The output of the method 245 is an updated configurations-set.

As mentioned above, step 255 includes steps 274 and 276, but not step278. Accordingly, the output of step 255 is the currentconfigurations-set with all configurations that may be reached from theconfigurations-set without consuming the character or in other words viaepsilon transitions. This assures that when the string matches thepattern defined by the backref-regex, a configuration associated with afinal state will be included into the configurations-set. For somebackref-regexes, the final state of backref-NFA may only reachable withepsilon transaction, such as shown in FIG. 5B.

FIG. 3 depicts a high-level block diagram of a method for estimating aworst-case complexity of a rule/pattern defined by a backref-regex,according to one embodiment. In general, a method 300 for estimating aworst-case complexity performs static analysis on a backref-NFA derivedfrom a particular backref-regex to provide better bounds for theworst-case space and time complexity. In one embodiment, the computedbounds are used as an admissibility test, to separate out rules/patternsdefined by regexes, which are potentially vulnerable to attacks. Suchrules may be subject to more detailed manual analysis.

The method 300 starts at step 305 and proceeds to step 310, where abackref-regex defining a particular rule/pattern is received. At step315, a corresponding backref-NFA is constructed in a manner describedabove with respect to FIG. 2A. Such a backref-NFA is analyzed at step320 to determine back-references alive at each NFA-state of thebackref-NFA.

According to one embodiment, not each back-reference entry in aconfiguration associated with a particular NFA-state is useful forfurther matching. Consider, for example, expression ‘(ab)\1(cd)\2’ shownin FIG. 5A. After the back-reference ‘\1’ has been used to match againstan input string, this back-reference is no longer needed for furthermatching. In particular, in the backref-regex of FIG. 5B, syntactically,there are two back-references and ‘\1’ is “dead” (i.e., unnecessary)after it has been used to match for the first time. Further, inbackref-regex, such as ‘(ab)(cd)\1;R;\2,’ the match for \1 is not usedin R while might be a large and highly ambiguous regular expression.

Also, for backref-regex with disjoint expressions, not eachback-reference is used to match the input string. For example, inexpression ‘(ab)\1|(cd)\2’ shown in FIG. 5B the back-references ‘\1’ and‘\2’ are used along disjoint union expressions, or in other words, alongalternative paths of the backref-NFA and at most, only one path would beselected during run of the backref-regex.

According to one embodiment, at step 320, back-references that areguaranteed to be “alive” (i.e., useful) at each NFA-state are determinedat compile-time. To determine such back-references, a liveness compilerprocess, such as the one that is used to determine variables that arestill in use at a program point, is adapted for analyzing thebackref-NFA structure for live back-references. A description of theliveness compiler process may be found in, for example, AHO, section9.2.5, pages 608-610, hereby incorporated by reference herein in itsentirety. The adaptation of this process for step 320 of the method 300,according one embodiment, is the following:

-   -   instead of a “program flow graph”, a graph of the backref-NFA is        used    -   instead of a “basic block”, backref-NFA edges are used    -   instead of variables, back-references are tracked. As discussed        above, an edge labeled with ‘\b’ denotes a “use” of        back-reference ‘b,’ while the edges labeled with ‘(_(b)’ and        ‘)_(b)’ form a definition of back-reference ‘b.’

Applying the adapted liveness process to the backref-NFA, a set ofback-references live at each NFA-state of the backref-NFA is generated.For example, for the backref-NFA FIG. 5A, the resulting sets of liveback-references are the following: q₄={1}, q₉={2}, and sets for theremaining NFA-states are empty. For the backref-NFA FIG. 5B, theresulting sets of live back-references are the following: q₄={1},q₁₀={2}, and sets for the remaining NFA-states are empty.

At step 325, the maximum number of live references over all states isdetermined. For example, for the both backref-NFAs of FIGS. 5A and 5Bsuch a number is 2. In one embodiment, at step 330, such information isused to determine the worst-case time complexity. For example, if L isthe maximum number of live back-references over all NFA-states of thebackref-NFA, then the worst-case time complexity can be tightened to O(m*K*m^(2L)), where K represents a number of states of the backref-NFAand m is the length of the input string or a packet. The method ends atstep 335.

In one embodiment, the matching process is optimized based on theresults of the adapted liveness process. For a configuration (q, m, s),entries in m which refer to back-references that are not alive atNFA-state q are dropped to obtain a smaller configuration (q, m′, s).This reduction may result in two configurations, which differ only withrespect to dead back-references being merged into a singleconfiguration, and thus, the size of the current set is reduced. Thatis, in various embodiments the configurations/configuration-setsgenerated during the matching process are reduced in size (i.e.,simplified) by removing those back-reference entries that are not alive.

In one embodiment, the worst-case time complexity value is furtherupdated. More specifically, when the backref-NFA for a pattern/rule hasa structure such that the matching process is purely deterministic, thematch complexity bound may be reduced. For example, consider thebackref-regex of the following form: ‘(a*)b(c*)d;R;\1\2,’ where R is anarbitrary regular expression. Assume that A₁ represents ‘(a*)b(c*)d,’and A₂ represents ‘R;\1\2.’ Then, though there are two liveback-references in this pattern, a sub-string matching each of theback-references is chosen uniquely. Therefore, there is a unique map,rather than the potential of m⁴ possible maps with the two liveback-references. Accordingly, the worst-case time complexity value maybe reduced to O (m*K), which is lower than the bound that would beobtained through simple liveness analysis.

In yet another embodiment, worst-case time complexity values arecalculated for all rules/patterns, rule-by-rule, and are used todetermine potentially “bad” rules, e.g., rules requiring unreasonablylong time to match against worst-case scenario malicious packets. Suchrules may be detected automatically and flagged for review by a rulewriter. In one embodiment, one or more pre-defined thresholds are usedto isolate the potentially “bad” rules. Such thresholds are compared tothe determined worst-case time complexity values. In this manner, therule writer is warned about potentially “bad” rules, the “bad” areisolated, and thus, robust performance in the face of malicious packetsis facilitated. For example, consider a regex of the form“(R1)(R2)(R3)(R4)\1\2\3\4” where R1,R2,R3,R4 are sub-expressions and\1,\2,\3,\4 are back-references. For this regex, the algorithm willdetermine that all 4 back-references are live at some states in thebackref-NFA. If the threshold is set to be 2, this exceeds the pre-setthreshold, and a warning is optionally generated for the rule-writer.

A person skilled in the art would readily recognize that steps ofvarious above-described methods can be performed by programmedcomputers. The embodiments of the invention are intended to cover suchprogrammed computers. For example, FIG. 6 depicts a high-level blockdiagram of a computer suitable for use in performing the functionsdescribed herein. As illustrated in FIG. 6, system 600 includes aprocessor element 620 (e.g., a central processing unit (CPU)); variousinput/output devices 610 (e.g., storage devices, including but notlimited to, a tape drive, a floppy drive, a hard disk drive, flashdrive, or a compact disk drive, a receiver, a transmitter, a speaker, adisplay, an output port, and a user input device (such as a keyboard, akeypad, a mouse, and the like)); a packets/rules processing module 630;and memory 640, e.g., random access memory (RAM) and/or read only memory(ROM). The packets/rules-processing module 630 includes a rulecharacterization engine 632 and a packet matching engine 634. As shown,the packet/rules-processing module 630 is adapted to access a set ofpacket matching rules 636. In one embodiment, the packet matching rulesare stored in a database, which may be part of the system 600, oralternatively, be separate from the system 600 and communicativelyaccessible by the system 600.

In general, the rule characterization engine 632 is adapted to estimatea worst-case complexity of a particular rule, such as one of the packetmatching rules 636. In one embodiment, the rule characterization engine632 estimates the worst-case complexity for the rule in accordance withthe method 300 described above with respect to FIG. 3. Thepacket-matching engine 634 is adapted to compare incoming packetsagainst one or more rules and determine whether the packet should berejected. In one embodiment, the packet matching engine 634 compares theincoming packets against the rules 636 in accordance with the method 200described above with respect to FIGS. 2A and 2B. Though both engines areshown as being included in the packets/rules-processing module 630, inone embodiment, only one of the engines is included into thepackets/rules-processing module 630. Furthermore, in one embodiment, therule characterization engine 632 and packet-matching engine 634 areplaced in two different computing devices. One skilled in the art wouldrecognize that other various implementations are possible.

In one embodiment, at least some of the above-described functions areimplemented in software and/or in a combination of software andhardware, e.g., using application specific integrated circuits (ASIC), ageneral purpose computer, or any other hardware equivalents. Thepackets/rules processing module 630 may be loaded into the memory 640and executed by the processor 620 to implement the functions asdiscussed above. As such, software instructions adapted for implementingthe packets/rules processing module 630 (including associated datastructures) or its various components may be stored on a tangiblecomputer readable medium (e.g., RAM memory, magnetic or optical drive ordiskette, flash drive, and the like), wherein the software instructionswhen executed by a processor/computer cause the processor/computer toperform the various functions discussed herein with respect to thevarious embodiments.

Furthermore, it is contemplated that some of the steps discussed hereinmay be implemented within hardware, for example, as circuitry thatcooperates with the processor to perform various method steps. Portionsof the functions/elements described herein may be implemented as acomputer program product wherein computer instructions, when processedby a computer, adapt the operation of the computer such that the methodsand/or techniques described herein are invoked or otherwise provided.Instructions for invoking methods described herein may be stored infixed or removable media and/or stored within a memory within acomputing device operating according to the instructions.

Moreover, the embodiments of the invention are also intended to coverprogram storage devices, e.g., digital data storage media, which aremachine or computer readable and encode machine-executable orcomputer-executable programs of instructions, where such instructionsperform some or all of the above-described methods and/or steps. Theprogram storage devices include, but are not limited to, digitalmemories, magnetic storage media such as magnetic discs or tapes, harddrives, or optically readable data storage media.

A powerful and expressive rule language is necessary to allow an IDS/IPSsystem to defend the network against complex new attacks. But this powercomes at a high cost, especially because the rule writer is typicallyunaware of the performance consequences of matching packets againstcomplex rules. Poorly written or incorrect rules can be very hard tomatch efficiently. Embodiments of the invention allow IDS/IPS itself toanalyze new rules and ensure that there are no resulting performancevulnerabilities, even in the presence of worst-case packet inputs. Inparticular, a deterministic matching process is described, where suchprocess allows analyzing rules, rule-by-rule, for the worst-casematching performance and exposing the few rules that are suspected tohave performance vulnerabilities. Moreover, techniques that provideon-the-fly deterministic matching of input strings against rules(patterns) defined by regular expressions with back-references aredescribed. Such techniques do not compromise the complexity of thelanguage used to describe the rules while provide guaranteedperformance.

Although various embodiments which incorporate the teachings of thepresent invention have been shown and described in detail herein, thoseskilled in the art can readily devise many other varied embodiments thatstill incorporate these teachings.

What is claimed is:
 1. A non-transitory computer readable medium storingsoftware instructions which, when executed by a processor, cause theprocessor to perform a method for estimating worst-case time complexityof a regular expression comprising one or more back-references(backref-regex), the method comprising: constructing a non-deterministicfinite automaton (NFA) corresponding to the backref-regex (backref-NFA),wherein the backref-NFA comprises a plurality of NFA-states and arespectively labeled edge for each of the one or more back-references ofthe backref-regex; performing liveness analysis on the backref-NFA todetermine for each NFA-state of the backref-NFA a set of back-referencesalive at the NFA-state; and determining a maximum number of aliveback-references over the plurality of NFA-states, wherein the determinedmaximum number is indicative of the worst-case time complexity of thebackref-regex.
 2. The medium of claim 1, wherein the method furthercomprises: determining a value of worst-case time complexity of thebackref-regex based on the determined maximum number; and providing thedetermined time-complexity value.
 3. The medium of claim 2, wherein thebackref-NFA further comprises, for each pair of brackets in thebackref-regex, a corresponding pair of edges labeled to identifyrespectively a beginning and an ending bracket, the edges extending froma sub-backref-NFA corresponding to a sub-backref-regex embedded insidethe pair of brackets.
 4. The medium of claim 3, wherein the edges of thebackref-NFA corresponding to the brackets of the backref-regex representepsilon transitions.
 5. The medium of claim 2, wherein constructing thebackref-NFA comprises: for each sub-backref-regex embedded in a pair ofmatching brackets, constructing a corresponding sub-backref-NFA, andadditional edges on both sides of the sub-backref-NFA, the additionaledges respectively labeled to identify an opening bracket and a closingbracket of the pair of matching brackets; and for each back-reference,constructing an additional edge labeled to indicate the back-reference.6. The medium of claim 1, wherein the method further comprises:comparing a value of the worst-case time complexity indicated by thedetermined maximum number to a pre-defined threshold; and if theindicated worst-case time complexity value is above the pre-definedthreshold, flagging the pattern as a potentially bad pattern.
 7. Themedium of claim 1, wherein the method further comprises removing fromsaid backref-NFA those NFA-state entries that refer to back-referencesthat are not alive.
 8. The medium of claim 2, wherein the method furthercomprises: comparing the determined worst-case time complexity to apre-defined threshold; and if the determined worst-case time complexityvalue is above the pre-defined threshold, flagging the pattern as apotentially bad pattern.
 9. The medium of claim 1, wherein the methodfurther comprises selecting a pattern for use in identifying maliciouspackets if the indicated worst-case time complexity associated with thepattern is below a pre-defined threshold.
 10. The medium of claim 9,wherein said method is used to process each of a plurality ofpattern-defining regular expressions to provide thereby a set ofpatterns selected for use in identifying malicious packets.
 11. Themedium of claim 9, wherein a received packet is identified as amalicious packet when an input string representing the received packetcompares favorably with a pattern selected for use in identifyingmalicious packets.
 12. The medium of claim 9, wherein said methodfurther comprises comparing an input string comprising a plurality ofcharacters representing received packet to a pattern selected for use inidentifying malicious packets via a single pass of a non-deterministicfinite automation (NFA) corresponding to the backref-regex (backref-NFA)to determine whether the input string matches the pattern.
 13. Themedium of claim 10, wherein said method further comprises forwardingpatterns selected for use in identifying malicious packets toward apacket-matching engine adapted to determine whether incoming packets therejected.
 14. The medium of claim 9, wherein said method furthercomprises comparing an input string representing a received packet to apattern defined by a regular expression containing one or moreback-references (backref-regex) via a single pass of a non-deterministicfinite automaton (NFA) corresponding to the backref-regex (backref-NFA),the input string comprising a plurality of characters, wherein thecomparing comprises: selecting sequentially characters of the inputstring until determining that the input string matches the pattern; andupdating, for each selected character, a configurations-set associatedwith the selected character, the configurations-set comprising one ormore configurations, wherein: the updating is based on the one or moreconfigurations, the selected character, and the backref-NFA; each of theone or more configurations is associated with a particular NFA-state ofthe backref-NFA and is adapted to indicate whether a particularback-reference of the backref-NFA is being matched; and the updatedconfigurations-set is associated with a next selected character on thecondition that the input string contains an unselected character;updating the configurations-set, after all characters of the inputstring have been selected, thereby generating a finalconfigurations-set; and determining whether the final configurations-setcontains a configuration associated with a final NFA-state.
 15. Themedium of claim 9, wherein said method is implemented within anintrusion detection system and intrusion prevention system (IDS/IPS)operative to analyze packets transiting between an unsecured network anda secured network.
 16. An apparatus, comprising: a memory; and aprocessor operably coupled to the memory and configured by instructionsstored thereon to estimate worst-case time complexity of a regularexpression comprising one or more back-references (backref-regex), theregular expression defining a pattern adapted for identifying maliciouspackets, the apparatus comprising a processor configured for:constructing a non-deterministic finite automaton (NFA) corresponding tothe backref-regex (backref-NFA), wherein the backref-NFA comprises aplurality of NFA-states and a respectively labeled edge for each of theone or more back-references of the backref-regex; performing livenessanalysis on the backref-NFA to determine for each NFA-state of thebackref-NFA a set of back-references alive at the NFA-state; determininga maximum number of alive back-references over the plurality ofNFA-states, wherein the determined maximum number is indicative of theworst-case time complexity of the backref-regex; and if the indicatedworst-case time complexity is below a pre-defined threshold, selectingthe pattern for use in identifying malicious packets.
 17. The apparatusof claim 16, wherein said apparatus is implemented within an intrusiondetection system and intrusion prevention system (IDS/IPS) operative toanalyze packets transiting between an unsecured network and a securednetwork.
 18. A tangible and non-transitory computer program productwherein computer instructions, when executed by a processor in a telecomnetwork element, adapt the operation of the telecom network element toprovide a method for evaluating a pattern to determine if the patternshould be included within a set of patterns used by malicious packetdetector, the malicious packet detector comparing an input stringrepresenting a packet received at a network element to one or morepatterns within the set of patterns to determine thereby whether saidpacket is malicious, the method comprising: estimating worst-case timecomplexity of a regular expression comprising one or moreback-references (backref-regex), the regular expression defining apattern adapted for identifying malicious packets; and if the estimatedworst-case time complexity is below a pre-defined threshold, selectingthe pattern for inclusion within the set of patterns; wherein saidestimating comprises: constructing a non-deterministic finite automaton(NFA) corresponding to the backref-regex (backref-NFA), wherein thebackref-NFA comprises a plurality of NFA-states and a respectivelylabeled edge for each of the one or more back-references of thebackref-regex; performing liveness analysis on the backref-NFA todetermine for each NFA-state of the backref-NFA a set of back-referencesalive at the NFA-state; and determining a maximum number of aliveback-references over the plurality of NFA-states, wherein the determinedmaximum number is indicative of the worst-case time complexity of thebackref-regex.
 19. The computer program product of claim 18, whereinsaid method further comprises forwarding patterns selected for use inidentifying malicious packets toward a packet-matching engine adapted todetermine whether incoming packets the rejected.
 20. The computerprogram product of claim 18, wherein said method further comprisescomparing an input string comprising a plurality of charactersrepresenting received packet to a pattern selected for use inidentifying malicious packets via a single pass of a non-deterministicfinite automation (NFA) corresponding to the backref-regex (backref-NFA)to determine whether the input string matches the pattern.